As the internet has increased in popularity, indexing the internet has become increasingly more important. One method of indexing the internet is through the use of a web crawler. A web crawler is a ‘bot’ or software application that visits websites and indexes or records the contents of the websites visited.